{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "deletable": true,
    "editable": true,
    "id": "4f3CKqFUqL2-",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Trying out features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Learning Objectives:**\n",
    "  * Improve the accuracy of a model by adding new features with the appropriate representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The data is based on 1990 census data from California. This data is at the city block level, so these features reflect the total number of rooms in that block, or the total number of people who live on that block, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "deletable": true,
    "editable": true,
    "id": "6TjLjL9IU80G"
   },
   "source": [
    "## Set Up\n",
    "In this first cell, we'll load the necessary libraries."
   ]
  },
  {
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
 "!sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst"
   ]
  },
  {

"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
 "# Ensure the right version of Tensorflow is installed.\n",
 "!pip freeze | grep tensorflow==2.5"
 ]
},
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import shutil\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "print(tf.__version__)\n",
    "tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)\n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = '{:.1f}'.format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "deletable": true,
    "editable": true,
    "id": "ipRyUHjhU80Q"
   },
   "source": [
    "Next, we'll load our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"https://storage.googleapis.com/ml_universities/california_housing_train.csv\", sep=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "deletable": true,
    "editable": true,
    "id": "HzzlSs3PtTmt",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Examine and split the data\n",
    "\n",
    "It's a good idea to get to know your data a little bit before you work with it.\n",
    "\n",
    "We'll print out a quick summary of a few useful statistics on each column.\n",
    "\n",
    "This will include things like mean, standard deviation, max, min, and various quantiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "test": {
      "output": "ignore",
      "timeout": 600
     }
    },
    "colab_type": "code",
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "id": "gzb10yoVrydW",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, split the data into two parts -- training and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(seed=1) #makes result reproducible\n",
    "msk = np.random.rand(len(df)) < 0.8\n",
    "traindf = df[msk]\n",
    "evaldf = df[~msk]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training and Evaluation\n",
    "\n",
    "In this exercise, we'll be trying to predict **median_house_value** It will be our label (sometimes also called a target).\n",
    "\n",
    "We'll modify the feature_cols and input function to represent the features you want to use.\n",
    "\n",
    "Hint: Some of the features in the dataframe aren't directly correlated with median_house_value (e.g. total_rooms) but can you think of a column to divide it by that we would expect to be correlated with median_house_value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def add_more_features(df):\n",
    "  # TODO: Add more features to the dataframe\n",
    "  return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create pandas input function\n",
    "def make_input_fn(df, num_epochs):\n",
    "  return tf.compat.v1.estimator.inputs.pandas_input_fn(\n",
    "    x = add_more_features(df),\n",
    "    y = df['median_house_value'] / 100000, # will talk about why later in the course\n",
    "    batch_size = 128,\n",
    "    num_epochs = num_epochs,\n",
    "    shuffle = True,\n",
    "    queue_capacity = 1000,\n",
    "    num_threads = 1\n",
    "  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Define your feature columns\n",
    "def create_feature_cols():\n",
    "  return [\n",
    "    tf.feature_column.numeric_column('housing_median_age')\n",
    "    # TODO: Define additional feature columns\n",
    "    # Hint: Are there any features that would benefit from bucketizing?\n",
    "  ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Create estimator train and evaluate function\n",
    "def train_and_evaluate(output_dir, num_train_steps):\n",
    "  # TODO: Create tf.estimator.LinearRegressor, train_spec, eval_spec, and train_and_evaluate using your feature columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "OUTDIR = './trained_model'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Run the model\n",
    "shutil.rmtree(OUTDIR, ignore_errors = True) # start fresh each time\n",
    "tf.compat.v1.summary.FileWriterCache.clear() \n",
    "train_and_evaluate(OUTDIR, 2000)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "first_steps_with_tensor_flow.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
